Coded light

ABSTRACT

A device comprising: an input for receiving a signal from a rolling-shutter camera which captures frames by exposing a lines of each frame in sequence, the signal comprising a coded light signal; and a signal processing module arranged to obtain a respective sample of the coded light signal from each of a number of said lines, being a substantially smaller number of the lines than exposed by the camera in each frame. The coded light signal comprises a message having a duration longer than this number of lines, and the message is repeated multiple times with a timing such that a different part of the message is seen by the camera in each of a plurality of different ones of said frames. The signal processing module is configured to time align the different parts of the message from the plurality of different frames, and reassemble the message from the time-aligned parts.

CROSS-REFERENCE TO PRIOR APPLICATIONS

This application is the U.S. National Phase application under 35 U.S.C.§ 371 of International Application No. PCT/EP2015/052461, filed on Feb.6, 2015, which claims the benefit of European Patent Application No.14155285.1, filed on Feb. 14, 2014 and European Patent Application No.14155269.5, filed on Feb. 14, 2014. These applications are herebyincorporated by reference herein.

TECHNICAL FIELD

The present disclosure relates to the communication of coded lightsignals embedded in the light emitted by a light source.

BACKGROUND

Coded light refers to a technique whereby data is modulated into thevisible illumination emitted by a light source, e.g. by an LED basedluminaire. Thus in addition to providing illumination (for which purposea light source may already be present in an environment), the lightsource also acts as a transmitter capable of transmitting data to asuitable receiver of coded light. The modulation is typically performedat a high enough frequency that it is imperceptible to human vision,i.e. so the user only perceives the overall illumination and not theeffect of the data being modulated into that illumination. In this waythe data may be said to be embedded into the light from the lightsource.

Coded light can be used in a number of applications. For instance, oneapplication is to provide information from a luminaire to a remotecontrol unit for controlling that luminaire, e.g. to provide anidentifier distinguishing it amongst other such luminaires which theremote unit can control, or to provide status information on theluminaire (e.g. to report errors, warnings, temperature, operating time,etc.). In one such example, the remote control unit may comprise amobile user terminal such as a smart phone or tablet having an inbuiltcamera. With the terminal running a suitable application, the user candirect the camera at a luminaire and thereby detect the identifier codedinto the light from that luminaire. Given the identifier of theluminaire it is looking at, the terminal may then control that luminaireby sending back a return signal (e.g. via RF).

In another application the coded light may be used to provideinformation to a user, e.g. to provide identifiers of the luminaires foruse in commissioning, or to enable provision of location relatedinformation. For example each luminaire in an indoor and/or outdoorenvironment (e.g. in the rooms and corridors of an office complex,and/or paths of a campus) may be arranged to emit light embedded with arespective identifier identifying it within that environment. If a userhas a mobile terminal equipped with a camera, and an associatedapplication for detecting coded light, the terminal can detect theidentifier of a luminaire illuminating its current location. This canthen be used to help the user navigate the environment, by looking upthe current location in location database mapping the identifiers tolocations of the luminaires. Alternatively or additionally, this may beused to look up information associated with the user's current location,such as information on exhibits in particular rooms of a museum. E.g.the look up may be performed via the Internet or a local network towhich the terminal has access, or from a local database on the userterminal. Alternatively the information could be directly coded into thelight from one or more luminaires. Generally speaking, the applicabilityof coded light is not limited.

Data is modulated into the light by means of a technique such asamplitude keying or frequency shift keying, whereby the modulatedproperty (e.g. amplitude of frequency) is used to represent channelsymbols. The modulation typically involves a coding scheme to map databits (sometimes referred to as user bits) onto such channel symbols. Anexample is a conventional Manchester code, which is a binary codewhereby a user bit of value 0 is mapped onto a channel symbol in theform of a low-high pulse and a user bit of value 1 is mapped onto achannel symbol in the form of a high-low pulse. Another example is therecently developed Ternary Manchester code, described in internationalpatent application publication no. WO2012/052935.

Ternary Manchester now forms a part of the state of the art and is thusknown to skilled person, but it is summarized again here forcompleteness. At the transmitter, each data bit to be transmitted ismapped to a channel symbol in the form of a respective unit pulse.According to this scheme, there are two possible units, in the form ofpositive and negative “hat” functions as shown in FIG. 5. The pulsemapped to a data bit of value 1 is shown on the left hand side of FIG.5, and the pulse mapped to a data bit of value 0 is shown on the righthand side of FIG. 5. A data bit is a bit of actual information to betransmitted, sometimes referred to as “user data” (even if notexplicitly created by a user). The data bit period is labeled T_(D) inFIG. 5, with the boundaries between user bit periods shown with verticaldashed lines.

Each unit pulse comprises a sequence of elementary channel periods oflength T_(C) in time, smaller than the data bit period. Each elementarychannel period conveys just one of the elementary levels that the codedsignal can take (one ternary Manchester symbol), and is not alonesufficient to convey data without being modulated into a compositechannel symbol. Hence each pulse of length T_(D) is the smallest or mostfundamental unit of information content that can be conveyed using thecoding scheme in question.

In the ternary Manchester code, each unit hat function comprises asequence of three elementary channel periods of length T_(C) in time,each half the length of the data bit period T_(D) (T_(D)=2T_(C)). Thethree elementary periods for a respective data bit are contiguous, withthe middle of the three being located at the center of the respectivedata bit period, so that the adjacent first and third elementary channelperiods straddle the beginning and end boundaries of the data bit periodrespectively by half an elementary channel period T_(C) either side.

For a data bit of value 1, this is mapped to the positive hat functionshown on the left of FIG. 5. The positive hat function comprises: afirst elementary channel period of height−½ centered on the beginning(earlier) boundary of the respective data bit period, followed by second(middle) elementary channel period of height+1 being centered on therespective data bit period, followed by a third elementary channelsymbol of height−½ centered on the end (later) boundary of therespective data bit period. The “height” at this stage may berepresented in any suitable terms such as a dimensionless digital value(ultimately to be represented by the modulated signal property, e.g.amplitude or frequency).

For a data bit of value 0, this is mapped to the negative hat functionshown on the right of FIG. 5. The negative hat function comprises: afirst elementary channel period of height+½ centered on the beginning(earlier) boundary of the respective data bit period, followed by second(middle) elementary channel period of height−1 being centered on therespective data bit period, followed by a third elementary channelperiod of height+½ centered on the end (later) boundary of therespective data bit period.

To create the coded bit stream to be transmitted, the hat functions ofadjacent user bits are added to one another, offset by the times oftheir respective bit periods. Because the hat functions overlap acrossthe boundaries between data bit periods, the functions add in theoverlapping regions between adjacent data bits. That is, the hatfunctions are joined along the boundaries, so the earlier boundary A_(n)of one data bit period is joined with the later bit boundary A_(n+1) ofthe preceding adjacent data bit period, with the height of the signalbeing summed where the two adjacent pulses overlap. An example of aresulting sequence of channel symbols in the time domain is shown inFIG. 6.

Where two adjacent data bits are of value 1, this means the twooverlapping elementary channel periods of height−½ add to a height of−1. Where two adjacent data bits are of value 0, the two overlappingelementary channel periods of height+½ add to height+1. Where twoadjacent data bits are of different values, the two overlappingelementary channel periods of height+½ and −½ add to 0. Thus in thecoded stream, each user bit period (each unit pulse) takes the form ofeither a positive pulse of a rectangular wave when a user bit of value 1is sandwiched between two adjacent user bits of value 1, or a negativepulse of a rectangular wave when a user bit of value 0 is sandwichedbetween two adjacent user bits of value 0, or an uneven pulse of one orfour possible configurations with straight edges when at least one ofthe adjacent user bits is different.

In an equivalent variant, the mapping of data bit values 0 and 1 topositive and negative hat functions may be reversed.

The resulting signal (e.g. that of FIG. 6) is then converted into avariation in the modulated property of the signal output by thetransmitting light source (e.g. whether represented in terms ofamplitude or frequency). For example, elementary channel symbol −1 maybe represented by a low light output level, the elementary channelsymbol +1 may be represented by a high output light level, and theelementary channel symbol 0 may be represented by an intermediate lightlevel between the high and low.

The ternary Manchester code can be advantageous as it provides asmoother transition when the data bits change value than a conventionalManchester code, and results in a spectrum in the frequency domain thatis more suppressed around low frequencies where interference such asmains hum may occur. However, the applicability of the presentdisclosure is not limited to ternary Manchester and in other embodimentsother examples of suitable coding schemes may be used, e.g. aconventional (binary) Manchester code, or other conventional binary orternary lines codes.

There is a growing interest in using coded light in applications wherethe light from a light source is to be captured using a rolling shuttercamera, such as the cheap cameras often found in mobile phone devices. Arolling shutter camera scans the lines of the image one at a time,line-by-line (typically at a minimum of 18 k lines/s). As the lines arerecorded time-sequentially, and the codes in the light may also varytime-sequentially, additional processing is involved. Typically thesamples on a line are “integrated” or “condensed” into a single valueper line. Each line thus captures a sample of the signal at a differentmoment in time, enabling the coded light signal to be reconstructed.

SUMMARY

According to one aspect disclosed herein, there may be provided a devicecomprising: an output for controlling a light source to embed a codedlight signal into visible light emitted from the light source, and acontroller configured to generate the coded light signal. The light isto be received by a rolling-shutter camera which captures frames byexposing a plurality of lines of each frame in sequence, the camerahaving an exposure time with each line being exposed for the exposuretime. The controller is configured to generate the coded light signalaccording to a format whereby the coded light signal comprises at leastone message and the message is repeated multiple times with a timingsuch that, when samples of the coded light signal are obtained from asubstantially smaller number of said lines than exposed by the camera ineach frame and the message is longer than said number of lines, adifferent part of the message is seen by the camera in each of aplurality of different ones of said frames.

In embodiments, the message may be repeated such that the whole messagewill be seen over said plurality of frames.

The message may be of a duration longer than one frame.

The message may comprise one or more packets comprising different datacontent, wherein each of the packets of the message may be followed byan inter-packet idle period, and wherein the repetitions of the messagemay be separated by an inter-message idle period different than theinter-packet idle period.

The inter-packet idle period may be greater than or equal to theexposure time or a maximum anticipated value of the exposure time.

The inter-message idle period may be selected to obtain said timingwhereby a different part of the message is seen by the camera in each ofa plurality of different ones of said frames.

The exposure time may be less than or equal to ( 1/30)s, less than orequal to ( 1/60)s, or less than or equal to ( 1/120)s.

The at least one message may be formed of at least three packets permessage.

Each of the packets may be of a length less than or equal to 17 bitslong, less than or equal to 12 bits long, or less than or equal to 9bits long.

The packet length may be 9 bits, consisting of a byte of content and asynchronization bit.

The controller may be configured to encode the coded light signalaccording to a ternary Manchester modulation coding scheme whereby databits of the signal are represented by being mapped to ternary Manchestersymbols.

The inter-message idle period may have a duration of at least 4 of saidsymbols.

Each of the packets may be 19 of said symbols long, the inter-packetidle period may have a duration of 33 of said symbols, and theinter-message idle period may have a duration of 5 of said symbols.

The controller may be configured to encode the coded light signal with asymbol rate of said symbols being 1 kHz, 2 kHz or 4 kHz.

The controller may be configured to receive an indication of theexposure time from the camera via a back channel, and to adapt theformat of the message based on the exposure time.

The controller may be configured to perform said adaptation by selectingone of more parameters such that a different part of the message is seenby the camera in each of a plurality of different ones of said frames,and the one or more parameters may comprise: the inter-packet idleperiod, inter-message idle period, number of packets per message, and/orsymbol rate.

The controller may be configured to adapt the format by selectingbetween a plurality of different predetermined combinations of saidparameters.

Said number of lines may be less than or equal to 14% of the lines ofeach frame.

According to a further aspect disclosed herein, there may be provided asystem comprising the device having any of the above features, the lightsource, and the camera; the camera being positioned relative to thelight source such that said samples are obtained from the substantiallysmaller number of lines than exposed by the camera in each frame and themessage is longer than said number of lines.

According to a further aspect disclosed herein, there may be provided amethod comprising: controlling a light source to embed a coded lightsignal into visible light emitted from the light source, the coded lightsignal comprising at least one message; receiving the light at arolling-shutter camera which captures frames by exposing a plurality oflines of each frame in sequence, the camera having an exposure time witheach line being exposed for the exposure time; and obtaining samples ofthe coded light signal from a substantially smaller number of said linesthan exposed by the camera in each frame, the message being longer thansaid number of lines; wherein the coded light signal is generatedaccording to a format whereby the message is repeated multiple timeswith a timing such that a different part of the message is seen by thecamera in each of a plurality of different ones of said frames.

According to a further aspect disclosed herein, there may be provided acomputer program product comprising code embodied on a computer-readablestorage medium and configured so as when executed to perform operationsof: controlling a light source to embed a coded light signal intovisible light emitted from the light source, to be received by arolling-shutter camera which captures frames by exposing a plurality oflines of each frame in sequence, the camera having an exposure time witheach line being exposed for the exposure time; and generating the codedlight signal according to a format whereby the coded light signalcomprises at least one message and the message is repeated multipletimes with a timing such that, when samples of the coded light signalare obtained from a substantially smaller number of lines than exposedby the camera in each frame and the message is longer than said numberof lines, a different part of the message is seen by the camera in eachof a plurality of different ones of said frames.

According to a further aspect disclosed herein, there may be provided acoded light signal embedded into visible light emitted from the lightsource, to be received by a rolling-shutter camera which captures framesby exposing a plurality of lines of each frame in sequence, the camerahaving an exposure time with each line being exposed for the exposuretime; wherein: the coded light signal is formatted according to a formatwhereby the coded light signal comprises at least one message and themessage is repeated multiple times with a timing such that, when samplesof the coded light signal are obtained from a substantially smallernumber of lines than exposed by the camera in each frame and the messageis longer than said number of lines, a different part of the message isseen by the camera in each of a plurality of different ones of saidframes.

In embodiments the method, computer program and/or signal may be furtherconfigured in accordance with any of the features discussed in relationto the device above.

According to another aspect disclosed herein, there may be provided adevice comprising: an input for receiving a signal from arolling-shutter camera which captures frames of a given duration at agiven frame rate by exposing a plurality of lines of each frame insequence, the signal comprising a coded light signal; a signalprocessing module connected to said input and arranged to obtain, viasaid input, a respective sample of the coded light signal from each of anumber of said lines, said number of lines being exposed over aduration, and said number being a substantially smaller number of saidlines than the plurality of lines exposed by the camera in each frame;the coded light signal comprises a message having a duration longer thanthe duration over which said number of lines are exposed, and multiplerepetitions of the message are received with a timing such that adifferent part of the message is seen by the camera in each of aplurality of different ones of said frames; and the signal processingmodule is configured to time align the different parts of the messagefrom the plurality of different frames, and reassemble the message fromthe time-aligned parts.

In embodiments, the whole message may be seen over said plurality offrames.

In embodiments, the message may have a duration longer than one frame.

In embodiments, the signal processing module may be configured toperform said time-alignment based on the frame duration and messageduration.

In embodiments, the signal processing module may be configured toperform said time-alignment by: determining a timing reference periodthat is an integer multiple of the length of said message; andoffsetting the part of the message received in each successive frame bythe frame length with respect to its preceding frame, but wrappingaround to the beginning of said timing reference period beyond the endof said timing reference period.

In embodiments, each frame comprises active lines and inactive lines,said plurality of lines being the active lines, and the signalprocessing module may be configured to obtain a respective sample fromeach of a plurality of active lines of each frame including said numberof lines, thereby producing a frame signal having said frame duration;and the offsetting is performed by extending each frame signal to havethe duration of the timing reference period.

Said extending may be performed by adding zeros to the frame signal.

The signal processing module may be configured to discard one or moreskipped frames.

The signal processing module may be configured to generate a pluralityof reassembled versions of the message each based on a differentrespective subset of said message parts, and to perform asynchronization between a clock of said device and a clocking of saidcoded light signal based on a correlation between said versions of themessage.

Each of said samples may be taken by combining pixel values of therespective line.

Said number of lines may be less than or equal to 14% of the lines ofeach frame.

Each of said parts may be less than or equal to 3% of the message.

The number of lines from which said samples are obtained may excludelines comprising one or more pixels that are over exposed.

According to a further aspect disclosed herein, there may be provided areceiver comprising the device having any of the above features, and thecamera.

According to a further aspect disclosed herein, there may be provided asystem comprising the receiver, and the light source; the camera beingpositioned relative to the light source such that said samples areobtained from the substantially smaller number of lines than exposed bythe camera in each frame and the message is longer than said number oflines.

According to another aspect disclosed herein, there may be provided amethod comprising: receiving a signal from a rolling-shutter camerawhich captures frames by exposing a plurality of lines of each frame insequence, the signal comprising a coded light signal; and obtaining arespective sample of the coded light signal from each of a number ofsaid lines, said number of lines being exposed over a duration, and saidnumber being a substantially smaller number of said lines than theplurality of lines exposed by the camera in each frame; wherein thecoded light signal comprises a message having a duration longer than theduration over which said number of lines is exposed, and the message isrepeated multiple times with a timing such that a different part of themessage is seen by the camera in each of a plurality of different onesof said frames; and wherein the method further comprises time aligningthe different parts of the message from the plurality of differentframes, and reassembling the message from the time-aligned parts.

According to another aspect disclosed herein, there may be provided acomputer program product embodied on a computer-readable medium andconfigured so as when executed to perform operations of: receiving asignal from a rolling-shutter camera which captures frames by exposing aplurality of lines of each frame in sequence, the signal comprising acoded light signal; and obtaining a respective sample of the coded lightsignal from each of a number of said lines, said number of lines beingexposed over a duration, and said number being a substantially smallernumber of said lines than the plurality of lines exposed by the camerain each frame; wherein the coded light signal comprises a message havinga duration longer than the duration over which said number of lines isexposed, and the message is repeated multiple times with a timing suchthat a different part of the message is seen by the camera in each of aplurality of different ones of said frames; and wherein the code isfurther configured so as when executed to time align the different partsof the message from the plurality of different frames, and to reassemblethe message from the time-aligned parts.

In embodiments the method and/or computer program may be furtherconfigured in accordance with any of the features discussed in relationto the device above.

According to yet another aspect disclosed herein, there may be provideda Wiener filter for equalizing an effect of a first filter on an inputsignal which is subject to the first filter and to noise and/orinterference, wherein: the first filter is dependent on at least oneunknown quantity; and the Wiener filter is configured based on anaveraged representation of the first filter averaged over said at leastone unknown quantity, in place of a representation of the first filterbeing assumed to be known.

In embodiments, said averaged representation may comprise an average ofthe conjugate of the first filter.

Said averaged representation may comprise an average of: the firstfilter multiplied by its conjugate.

Said averaged representation may comprise an average of the conjugate ofthe first filter and an average of: the first filter multiplied by itsconjugate.

The Wiener filter may operate in a frequency domain.

The Wiener filter may be configured according to:

$G = \frac{{E_{\theta}\left\lbrack H^{*} \right\rbrack} \cdot S}{{{E_{\theta}\left\lbrack {HH}^{*} \right\rbrack} \cdot S} + \mathcal{N}_{0}}$where G is the Wiener filter in the frequency domain, H is the firstfilter in the frequency domain, S is a spectral density of the inputsignal, N₀ is a spectral density of the noise and/or interference, θ isthe unknown quantity, and E is the average with respect to θ.

The average may assume a uniform distribution of the unknown quantitybetween finite limits.

The first filter may have a nominal value, and said averaging withrespect to the unknown quantity may be computed using a Taylor seriesexpansion of the first filter around its nominal value and a firstplurality moments of the unknown quantity.

The first filter may be dependent on a plurality of unknown quantities,and the Wiener filter may be configured based on an averagedrepresentation of the first filter averaged over each of said unknownquantities.

The first filter may comprise a box function in the time domain and asinc function in the frequency domain, the box function having a widthin the time domain, and said unknown quantity may comprise the width ofthe box function.

The input signal may comprise a coded light signal captured by a rollingshutter acquisition process whereby each line of a frame is exposed inturn for an exposure time, and said filter may be a result of therolling shutter acquisition process with the exposure time being saidunknown quantity.

The exposure of each line may produce the box function, and its widthmay be the exposure time.

The first filter may comprise a band pass filter having a centerfrequency and band width, and said at least one unknown quantity maycomprise the center frequency and/or band width of the band pass filter.

According to a further aspect disclosed herein, there may be provided areceiver comprising the Wiener filter having any of the above features,and the camera which may be arranged to capture said input signal bysaid rolling shutter acquisition process.

According to a further aspect disclosed herein, there may be provided amethod of determining a Wiener filter for equalizing an effect of afirst filter on an input signal which is subject to the first filter andto noise and/or interference, the method comprising: identifying atleast one unknown quantity upon which the first filter is dependent; andin a formulation of a Wiener filter comprising a representation of thefirst filter, in place of a representation in which the first filter isassumed to be known, replacing the representation with an averagedrepresentation of the first filter averaged over said at least oneunknown quantity.

According to a further aspect disclosed herein, there may be provided acomputer program product embodied on a computer-readable medium, andconfigured so as when executed to implement a Wiener filter forequalizing an effect of a first filter on an input signal which issubject to the first filter and to noise and/or interference, wherein:the first filter is dependent on at least one unknown quantity; and theWiener filter is configured based on an averaged representation of thefirst filter averaged over said at least one unknown quantity, in placeof a representation of the first filter being assumed to be known.

In embodiments the method and/or computer program may be furtherconfigured in accordance with any of the features discussed in relationto the Wiener filter above.

In further embodiments, any of the features of the transmit-side device,receive side device and/or Wiener filter set out above may be combined;as may any features of any device, transmitter, receiver, system,signal, method and/or computer program set out above or disclosedelsewhere herein.

Note that this Summary section is not intended to limit the scope of thepresent disclosure. The scope of the disclosure is limited only by theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

To assist understanding of the present disclosure and to show howembodiments may be put into effect, reference is made by way of exampleto the accompanying drawings in which:

FIG. 1 is a schematic block diagram of a coded light communicationsystem,

FIG. 2 is a schematic representation of a frame captured by a rollingshutter camera,

FIG. 3 shows an example of a captured frame,

FIG. 4 shows an example of usable samples obtained from a capturedframe,

FIG. 5 schematically illustrates a Ternary Manchester coding scheme,

FIG. 6 schematically illustrates an example signal coded using TernaryManchester,

FIG. 7 schematically illustrates a message format,

FIG. 8 is a graph of a number of frames required for message reassembly,

FIG. 9 shows three repetitions of a cyclically repeated message,

FIG. 10 shows a message consisting of three packets,

FIG. 11 shows transmitted and received signals in the time domain,

FIG. 12 shows a transmitted signal and ISI in the frequency domain,

FIG. 13 shows signals obtained by sampling each of a plurality offrames,

FIG. 14 shows extended versions of the signals obtained from samplingframes,

FIG. 15 shows time aligned versions of the signals from a plurality offrames,

FIG. 16 shows signals reconstructed from a plurality of frames,

FIG. 17 shows a reconstructed message,

FIG. 18 shows a non-rolling alignment of messages,

FIG. 19 shows a “switching” alignment of messages,

FIG. 20 is a schematic block diagram of a Wiener filter equalizing afilter H,

FIG. 21 is a schematic block diagram of an ISI filter resulting from arolling shutter,

FIG. 22 is a schematic block diagram representing additive noise, and

FIG. 23 is a schematic block diagram of a robust Wiener filterequalizing a filter H.

DETAILED DESCRIPTION OF EMBODIMENTS

The following relates to a coded light application, and provides aformat for transmitting coded light, a decoder for receiving the codedlight, and one particular advantageous building block used in thedecoder (which can also be used in other applications other than codedlight).

The format and decoding techniques are aimed at providing a practicalsolution for coded light, defining a format that can work with existingrolling-shutter cameras as well as dedicated so-calledregion-of-interest (ROI) cameras alike. The disclosure provides a methodof encoding and decoding, an encoder and decoder, a signal format, andsoftware for encoding and decoding, that in embodiments allow such cheaprolling shutter cameras to receive coded light and to decode themessages contained therein.

Cheap rolling shutter cameras effectively scan their images, so as thelines progress, so does time. This implies that the timestamp of the topline is much earlier than the timestamp of the bottom line. Now imaginethat coded light is present in the image—the coded light will typicallyonly be visible in a small section of the image.

The lines that actually image the light are the lines that contain codedlight. Each line is “condensed” into a single value and that singlevalue corresponds with a bit of information or a symbol; that is the bitor symbol transmitted at the moment in time that the line was scanned.Now for the rolling shutter camera to decode a message, one could makesure that the number of lines per frame is high enough (so the light bigenough) and decode images based on a single frame. However, as will bediscussed in more detail shortly, that is not always possible.

FIG. 1 gives a schematic overview of a system for transmitting andreceiving coded light. The system comprises a transmitter 2 and areceiver 4. For example the transmitter 2 may take the form of aluminaire (e.g. mounted on the ceiling or wall of a room, afree-standing lamp, or an outdoor light pole); and the receiver 4 maytake the form of a mobile user terminal such as a smart phone, tablet orlaptop computer. The transmitter 2 comprises a light source 10 and adriver 8 connected to the light source 10. The transmitter 2 alsocomprises a device in the form of a controller 6 and an output to thedriver 8, for controlling the light source 10 to be driven via thedriver 8. For example the controller 6 may take the form of softwarestored on a memory of the transmitter 2 and arranged for execution on aprocessor of the transmitter, or alternatively it is not excluded thatsome or all of the controller 6 could be implemented in hardware, orconfigurable or reconfigurable hardware circuitry. The receiver 4comprises a camera 12 and a device in the form of a signal processingmodule 14 and input from the camera 12. The signal processing module 14may for example take the form of software stored on a memory of thereceiver 4 and arranged for execution on a processor of the receiver 4,or alternatively it is not excluded that some or all of the signalprocessing module 14 could be implemented in dedicated hardwarecircuitry, or configurable or reconfigurable hardware circuitry. Thecontroller 6 and signal processing module 14 are configured to performtransmit-side and receive side operations (respectively) in accordancewith embodiments disclosed herein.

Referring to FIGS. 2 and 3, the camera 12 is arranged to capture aseries of frames 16, which if the camera is pointed towards the lightsource 10 will contain an image of light from the light source 10. Thecamera 12 is a rolling shutter camera, which means it captures eachframe 16 not all at once (as in a global shutter camera), but byline-by-line in a sequence of lines 18. That is, each frame 16 isdivided into a plurality of lines 18 (total number of lines labeled 20),typically horizontal lines, each spanning across the frame 16 and beingone or more pixels thick (e.g. spanning the width of the frame 16 andbeing one or more pixels high in the case of horizontal lines). Thecapture process begins by exposing one line 18, then the next (typicallyan adjacent line), then the next, and so forth. For example thecapturing process may roll top-to-bottom of the frame 16, starting byexposing the top line, then the next line from top, then the next linedown, and so forth. Alternatively it could roll bottom-to-top (or evenin vertical lines side to side). Note that the exposures of each linemay be exclusive in time, or alternatively may overlap in time (buteither way begin at different times). The camera 12 has a certainexposure time Texp, and each line is exposed in turn for an instance ofthis same exposure time. Note also that in the case of digital cameras,“exposure” does not mean in the sense of a mechanical shutter, butrather the time for which the pixels of the line are capturing orsampling the light.

To capture a sample for the purpose of detecting coded light, some orall of the individual pixels samples of each given line 18 are combinedinto a respective combined sample 19 for that line (e.g. only the“active” pixels that usefully contribute to the coded light signal, tobe discussed later with reference to FIGS. 3 and 4). For instance thecombination may be performed by integrating or averaging the pixelvalues, or by any other combination technique. Alternatively a certainpixel could be taken as representative of each line.

In the existing literature it is assumed that the source 10 covers allor almost all of every frame. However this is often not the case.Moreover the light being emitted is not necessarily synchronized withthe capturing process which can result in further problems.

A particular problem in using a rolling shutter camera for coded lightdetection therefore arises, because the light source 10 serving as acoded light transmitter may in fact cover only a fraction of the lines18 of each frame 16. Actually, only the lines 24 in FIG. 2 containpixels that record the intensity variations of the coded light sourceand thus lead to samples containing useful information. See also FIG. 3.All the remaining “lines per frame” 22 and their derived samples do notcontain coded light information related to the source 10 of interest. Ifthe source 10 is small, one may only obtain a short temporal view of thecoded light source 10 in each frame 16 and therefore the existingtechniques only allow for very short messages. However, it may bedesirable to have the possibility of also transmitting longer messages.Note also that there may be some lines 26 that are “hidden” or inactive,e.g. due to a selected frame format (leaving only active lines 21contributing to the captured image).

Apart from the above there may alternatively or additionally be one ormore other problems. In embodiments problems may comprise: firstly, arolling shutter may result in short temporally-interrupted views of thecoded light source; secondly, there may be a conflict of interestbetween “automatic exposure control” and coded light; thirdly, drivertechnology at present allows only low frequency signaling which maycause flicker; and/or fourthly, the filtering effect produced by therolling shutter process may result in inter-symbol interference (ISI).

Therefore existing techniques may be insufficiently flexible and/orprone to error or interference. The following embodiments aim to combineinformation from multiple video frames in a rolling shutter camera, suchthat messages longer than their footprint in a single video frame can becaptured and decoded. In embodiments this involves:

(i) use of a signal format whereby a message is cyclically repeated bythe transmitter; and

(ii) at the receiver, exploiting the knowledge of the repetition time ofthe message (Tm) and the knowledge of the frame duration (Tframe) forreconstructing a complete message from the partial snapshots obtained ineach frame. To this end the disclosure provides a method to collect andreassemble the data collected from multiple frames.

A message is cyclically repeated, and at the receiver the message iseffectively re-assembled over time (e.g. this can for certain messagesactually take 1 or 2 seconds, e.g. so 30-60 frames). In order to enablethis, the following describes a particular data format for encodinginformation in the light.

Part of the decoding of the signal in turn is described using a methodreferred to herein as “reassembly”. To facilitate the decoding, themessage duration and/or the Texp of the camera are tweaked in a mannerthat enables a cheap rolling shutter camera to detect a complete messagefairly quickly.

Once the message is re-assembled it will be equalized. The “normal”approach is to take the message and to effectively use a slicer todetermine the exact timing of the signal and then equalize it. However,according to embodiments of the following, this can be achieved in asmart manner using a robust Wiener filter implementation that is ratherefficient (preferably such that the entire decoding algorithm can beimplemented on standard run-of-the-mill mobile phones).

The robust Wiener filter takes into consideration the uncertainty of thechannel and in this manner can reduce the inter-symbol interference(ISI). In the following embodiments this filter is used followingre-assembly, but note that it may be used in other systems as well (notlimited just to equalizing the effect of a rolling-shutter nor even justto coded light applications).

Message Format

The following describes a message format that allows for a reliablecombination of the information of multiple video frames such thatmessages longer than the “footprint”, and even messages having aduration of many frames can be captured and decoded. Moreover, thesignal format allows for asynchronous (Wiener-like) equalization to undothe ISI caused by the camera at the receiver. Further, the frequencycontent of the messages can be such that there is no visible flicker orstroboscopic effects, even for message lengths having a repetitionfrequency of, e.g., 18 Hz (very sensitive flicker frequency).

An example of such a message format is shown in FIG. 7. To ensure themessage can be captured even given a small footprint, the coded lightsignal is transmitted according to a format whereby the same message 28is repeated multiple times in succession, and the timing of this isconfigured relative to the exposure time of the camera—or the range ofpossible exposure times of anticipated cameras—such that the message“rolls” over multiple frames. That is, such that a different part of themessage is seen by the camera in each of a plurality of differentframes, in a manner that allows the full message to be built up overtime as different parts of the message are seen. The issue here istherefore the manner in which the message length (duration) Tm is chosenrelative to the exposure time Texp or anticipated exposure times, suchthat in reconstruction the rolling shutter camera images another part ofthe message in every frame (wherein the parts of the message are notnecessarily consecutive, and in fact for rolling shutter cameras theywill often not be consecutive). The message timing may be adapted inresponse to actual knowledge of a particular camera's exposure Texpbeing fed back via a suitable back channel such as an RF channel betweenreceiver 4 and transmitter 2 (a “negotiated format”), or alternativelythe timing may be formatted in a predetermined fashion to anticipate arange of possible exposure times values Texp of cameras the format isdesigned to accommodate (a “universal format”).

In embodiments, aside from the length (duration) of the message's actualdata content (payload) 30, the message length Tm may be selected byincluding an inter-message idle period (IMIP) 34 between repeatedinstances of the same message. That way, even if the message contentalone would result in each frame seeing more-or-less the same part ofthe message, the inter-message idle period can be used to break thisbehavior and instead achieve the “rolling” condition discussed above. Inembodiments the inter-message idle period may be adapted given feedbackof Texp (“negotiated format”), or may be predetermined to accommodate arange of possible values of Texp (“universal format”).

As mentioned, the rolling condition is linked to the exposure time (i.e.line exposure time) Texp of the rolling-shutter camera. There is no onesingle solution to this, it is more a matter of avoiding combinations ofTm and Texp that do not meet the condition (discussed in more detailshortly). In the case of seeking a universal format, the inventors havediscovered that sufficient solutions can be assured to be available aslong as Texp<=33 ms or ( 1/30)s (approximately).

Another issue is inter-symbol interference (ISI), which is a result ofthe filtering effect of the exposure of each line (effectively a boxfilter applied in the time domain as each line is exposed). To mitigatethis, in embodiments the message format is arranged such that eachinstance of the message comprises a plurality of individual packets 29(e.g. at least three) and includes an inter-packet idle period (IPIP) 32between each packet. In embodiments, the inter-packet idle periodfollows each packet, with the inter-message idle period (IMIP) 34 taggedon the end after the last packet (there could even be only one packet,with the IPIP 32 and potentially IMIP 34 following).

Inter-symbol interference is then a function of packet length andinter-packet idle period. The more data symbols there are in a row, themore inter-symbol interference (ISI). Therefore it is desirable to keepthe packet length small with good sized gaps in between. The idle gaps(no data, e.g. all zeros) between bursts of data helps to mitigate theinter-symbol interference, as does keeping the packet length short.Again these properties may be adapted in response to actual knowledge ofa particular camera's exposure time Texp being fed back via a suitableback channel such as an RF channel between receiver 4 and transmitter 2(“negotiated format”), or alternatively the timing may be formatted in apredetermined fashion to anticipate a range of possible exposure timevalues Texp of cameras the format is designed to accommodate (“universalformat”). In embodiments, the inventors have discovered that a packetlength no longer than 9 bits separated by an inter-packet idle period ofat least Texp provides good performance in terms of mitigating ISI. Byconvenient coincidence, 9 bits also advantageously allows for one byteof data plus a synchronization bit. Nonetheless, in other embodiments apacket length of up to 12 bits, or even up to 17 bits may be tolerated.

As well as achieving “rolling”, another potential issue issynchronization. The receiver has a template of the message format whichis uses to synchronize with the received signal—e.g. it knows that aftera gap of the IPIP+IMIP, to expect a synchronization bit, then a byte ofdata, then the IPIP, then another synchronization bit and byte of data,etc. By comparing this template with the received coded light signal,the receiver can synchronize with the signal. In embodiments, in orderto assist synchronization, the inventors have found that theinter-message idle period should preferably be at least 4 symbols of therelevant modulation code, e.g. 4 ternary Manchester symbols.

Given the above considerations, an exemplary message format comprises:

(i) use of a signal format where a message is cyclically repeated (manytimes) by the transmitter, thus allowing a (temporal) recombination offootprints from consecutive video frames, each footprint containing apartial received message, for obtaining a complete receivedmessage—message size may be chosen such that by cyclic repetitioneventually the entire message can be recovered;(ii) a message having relatively short packets (e.g. of 9 bits),separated by inter-packet idle periods for allowing an equalizer toreconstruct the original transmitted waveform in the presence of heavyISI caused by an un-controllable camera exposure time setting; and(iii) using a form of Ternary Manchester (TM) as a DC-free modulationcode, leading to extra suppression of low frequency components, thuseliminating flicker at low symbol frequencies.

Variations are also possible. For example, while the preferredmodulation code is ternary Manchester (which may be abbreviated by theinitials TM), other codes could alternatively be used (preferablyDC-free or low DC content, with no visible flicker), e.g. conventionalManchester or non-return to zero (NRZ). The following also furtherdescribes various particularly advantageous choices for the formatparameters (e.g. IMIP). In further embodiments, the IPIP may be tuned tothe maximum exposure time. The TM-symbol length may also be tuned toexposure time when exposure time>IPIP. In yet further embodiments,guided descrambling may be used for medium length messages, and/orunscrambled short packets for short messages.

Returning to FIG. 2, some further details are now discussed. Asmentioned, existing literature assumes that the source to be decodedcovers almost or entirely every frame. It is assumed that the durationof a single message to be decoded is such that in can be captured in thefootprint of the source in a single frame. It is recognized that the“hidden lines” 26 can form a problem because of the synchronicitybetween the data packets and the shooting of the frames. It is suggestedthat a message may be repeated such that at least one repetitionsatisfies the condition of being captured completely within a singleframe. However, existing data formats for coded light can still sufferfrom a number of problems.

As already discussed, a particular problem in using a rolling shuttercamera for coded light detection arises because the light source servingas a coded light transmitter may cover only a fraction of the lines ofeach frame (see again FIG. 2). Actually, only the lines covering thesource contain pixels that record the intensity variations of the codedlight source. All the remaining lines and pixels do not contain codedlight information related to the source of interest. If the source issmall, one only obtains short, temporally-interrupted views of the codedlight source in each frame and therefore the existing techniques onlyallow for very short messages.

Another issue is that current smartphones such as iPhones and iPads donot allow for control of the exposure time Texp and ISO by an “app”.Existing automatic built-in control algorithms often lead to longexposure times that, after camera detection, lead to heavy inter-symbolinterference (ISI) between the digital symbols that are sequentiallytransmitted by the light source.

Further, current LED driver technology only allows for cheap,energy-efficient solutions if the bandwidth (symbol rate) of thetransmitted digital signal is very limited (say a symbol rate between 1and 8 kHz). For such low frequencies, flicker and stroboscopic effectsmay become serious, unless special precautions are taken in the signalformat for suppressing low frequencies. Having just a DC-free code doesnot always suffice.

The present disclosure describes a signal format that allows for areliable combination of the information of multiple video frames suchthat messages longer than the “footprint”, and even messages having aduration of many frames can be captured and decoded. Moreover, thesignal format allows for asynchronous (Wiener-like) equalization to undothe ISI caused by the camera at the receiver. Finally, the frequencycontent of the messages can be such that there are no visible flicker orstroboscopic effects, even for message lengths having a repetitionfrequency of, e.g., 18 Hz (very sensitive flicker frequency).

A snapshot of a typical coded-light signal at the transmitter isdepicted in FIG. 9, which is described next. It is assumed that thelight source can vary its (instantaneously) emitted light intensitybetween 0 and 1. In FIG. 9, the average light intensity (DC) is set to0.8, and the amplitude of the coded light signal equals 0.1. The codedlight signal is superimposed onto the average (DC) light level.

A message, in this example having a duration of 161 ms, consists of 3packets, each packet comprising 9 TM-encoded bits. A message iscyclically repeated by the transmitter (3 repetitions are shown in FIG.9). The TM-symbol rate equals 1 kHz (1000 TM-symbols per second).

Each packet of a message in this example is trailed by an inter-packetidle period of 33 TM-symbols (˜33 ms). At the end of each message, thereis an (extra) inter-message idle period of 5 TM-symbols, resulting in atotal idle period of 33+5=38 idle symbols between the third packet ofthe current message and the first packet of the next message. FIG. 9depicts 3 repetitions of a message, where each message consists of 3packets.

FIG. 10 depicts a single message of FIG. 9, where the DC has beenremoved and the amplitude of the signal has been made equal to 1. Theactive part of each packet consists of 9 TM-encoded bits, leading to2·9+1=19 TM symbols. Note that the first and the last TM symbol of eachpacket have an amplitude of ±0.5, consistent with TM encoding rules. Themessage format, as described in FIGS. 9 and 10, can be decoded using acamera that has any given Texp such that Texp≤ 1/30. In general allparameters such as e.g., TM-symbol rate, idle periods, modulation codemay be selected to facilitate detection.

The reason for cyclically repeating a message is that, at each frame ofa rolling shutter camera movie, only a small part of the transmittedmessage may be recoverable. The size of that part depends on the size ofthe light source in the images of the camera (footprint), and of theduration of the message. For instance, if the size of the light sourceis such that only 14% of the lines of a frame are covered by the lightsource, and if the duration of the message is in the order of 5 frames(assuming a recording speed of 30 frames/second), only about 3% of amessage is potentially recoverable from a single movie frame.

If the message duration is carefully chosen with respect to the framerate of the movie, consecutive frames of the movie reveal differentparts of the repeated message such that eventually the whole message isrecovered.

FIG. 8 depicts how the number of frames required for obtaining acomplete message, depends on the message duration and the size of thefootprint in the image for a frame rate of 29.97 fps.

The following considers the relationship shown in FIG. 8. For each frameof duration Tf, a view of duration Tfootprint of the message isobtained. A collection of N footprints of N consecutive frames has tocover at least 1 complete message. The footprints have to “roll” overthe messages. Footprints have a repetition frequency equal to the framerate (=29.97 Hz), messages have a repetition frequency of 1/Tm, andthese frequencies must be “sufficiently” different.

It may also be desired to minimize N, as a large N leads to largelatencies. Also for a “small” footprint, one may desire a small N, e.g.N=30˜1 second.

Transmitter frequency deviations lead to Tm variations. Some deviationsmay lead to “slow rolling” or even absence of rolling. N has to remainreasonable for a certain range of message durations around a nominalvalue.

Now consider what happens to covering a message with footprints if:

relative footprint α=Tfootprint/Tf=0.4

0<α≤1, (in practice e.g. 0<α≤0.88 due to hidden lines)

If Tm is about Tf, the message barely rolls (each frame sees practicallythe same part of the message). But if Tm is about 1.5 times Tf, themessage “switches” so that every other frame sees alternate parts of themessage, but some parts are repeatedly missed.

It turns out that, if α<1, one obtains “non-rolling” footprints if themessage durations Tm are a multiple of the frame duration Tf. If α<0.5,one obtains “switching” footprints if Tm is a half-integer multiple ofTf (0.5, 1.5, 2.5, . . . ).

In general, if 1/(n+1)<α≤1/n, where n is integer, then one encounters“non-rolling” footprints if:

$\left. {{{\frac{T_{m}}{T_{f}}\epsilon\left\{ \frac{k}{m} \right.m} = 1},\ldots\mspace{14mu},n,{k\;\epsilon\; N^{+}}} \right\}$

It turns out that the rolling may already be insufficient if the aboveratio is “close” to one of the “non-rolling” ratios. It also turns outthat the rolling may already be insufficient if the above ratio is“close” to one of the “non-rolling” ratios.

The result is a complicated relationship as seen in FIG. 8.

Modulation Code

The preferred modulation code for low bit rates is ternary Manchester(TM) because of the extra suppression of low frequency components thatmay lead to flicker. Low bit rates might be imperative because of tworeasons: (i) the limited affordable complexity and minimum requiredefficiency for drivers of the LED light sources; and/or (ii) forobtaining a signaling speed that can be recovered for very long exposuretimes.

Comparing NRZ, Manchester and ternary Manchester, note that NRZ(actually: no modulation code) has a very high DC content. TheManchester modulation code, well-known from magnetic recording, and alsoproposed for the IEEE Visible Light Communication (VLC) standard, is aso-called DC-free code, i.e., the spectral content at frequency zeroequals 0. The Ternary Manchester modulation code is a so-called DC²-freemodulation code, implying that the spectral density around DC remainsmuch smaller compared to a DC-free code like Manchester. In the spectrafor low frequencies, Tm is therefore advantageous compared toManchester. For flicker, frequencies up to 100 Hz are important.

Since the signal format makes use of relatively short packets,interspersed with idle symbols, one can guarantee a message to beDC²-free, by letting each packet be DC²-free. This is accomplished bymodulating the user bits using the TM impulse response {−0.5, 1, −0.5}.Note that a packet of 9 user bits leads to a TM-encoded packet of 19TM-symbols.

For larger bit rates, other modulation codes, maybe even multi-levelDC-free modulation codes (e.g. quaternary Manchester) also can beenvisioned, provided the spectral densities do not lead to visibleflicker.

The modulation codes to be used can be defined in a manner that allowsfor some freedom in the actual implementation of the driver, e.g. fordrivers having an Amplitude Modulation (AM) implementation or fordrivers having a Pulse Width Modulation (PWM) implementation. Thisimplies that, in contrast to traditional modulation formats, the actualshape of the waveforms to be transmitted is not exactly defined forcoded light.

A preferred way of defining a modulation code for coded light would beto define the rules and acceptable values of the output of a full-Tmoving-average filter applied to a modulator output waveform at theoptimum sampling points.

Packet Length

Turning to the question of packet length, the packet length ispreferably chosen such that the worst case data pattern is stillrecoverable under worst case exposure times.

An example is shown in FIG. 11. Consider a transmitted waveform 36corresponding to a TM-encoded packet of 9 bits consisting of all ones(fsymbol=1 kHz). If this waveform is detected by a camera having a Texp=1/125 [s], one can obtain a 1-dimensional received waveform 38 at theoutput of the camera, by proper processing of a sequence of videoframes. Note that the received signal, which is a distorted version ofthe transmitted signal, can be seen as would be generated by the camera,by convolving the transmitted signal by a rectangular box function,corresponding to a FIR filter action of Texp (moving average over Texpseconds).

The moving average filtering of Texp leads to inter symbol interference(ISI) between the TM-symbols of the packet. Note the reduction of theamplitude of the received signal with respect to the incomingtransmitted signal. Also note that in the last half of the packet theamplitude of the received signal has been reduced to zero. Finally notethat the received signal extends beyond the transmitted signal by Texp=8ms because of the causal FIR-type filtering by Texp. It is the task ofthe signal processing in the receiver to reconstruct the transmittedsignal from the received signal.

FIG. 12 shows the same situation in the frequency domain. The curve 40represents the absolute value of the spectral representation (theFourier transform) of a single 9-bit TM-encoded packet consisting offall ones. The curve 42 represents the absolute value of the transferfunction of the “Texp moving average filter”. The received signal in thefrequency domain is the dot product of both spectral representations.Note that the zeros of the ISI filter are particularly detrimental tothe received signal, since the signal at those frequencies (and at thefrequencies in the neighborhood of the zeros) effectively are removedfrom the transmitted signal.

If one desires that the transmitted signal is recoverable from thereceived signal, it is required that at least sufficient signal energyremains after filtering the transmitted signal with the ISI filter forall reasonable choices of Texp. For this to happen, the spectralrepresentation of the transmitted signal has to be sufficiently “spread”across many frequencies (for all possible choices of the bit content ofa packet). This turns out to be the case if the packet length is in theorder of 9 bit.

On the other hand, if one would make a packet (consisting of all ones)longer than 9 bits (say 17 bits), the spectral representation of such along packet would still be concentrated around 500 Hz, but its spectralwidth would be about ½ of the original packet. It turns out that in thatcase too much signal energy is destroyed by the ISI filter.

The inventors have found that, using TM modulation with fsymbol=1 kHz,for a packet length from say 9 to 12 bits, one can recover thetransmitted signal sufficiently accurately for all Texp≤( 1/30)s,provided the inter-packet idle period (IPIP) is at least Texp. Notethat, if IPIP=( 1/30)s, a fixed transmit signal format works for allTexp≤( 1/30)s. This may be used to provide a universal signal format.

If the packet length is between 12 and 17 bits long, it turns out thatthe minimum eye height of the eye pattern is determined by only a few“detrimental” bit patterns that have a poor spectral representation thatcan be destroyed by the “Texp moving average filter” in such a mannerthat it is irrecoverable. If those detrimental bit patterns are only afew, one can avoid those from occurring by so-called “guidedscrambling”. However, it turns out that one requires in the order of 16different scrambling patterns for applying a useful guided scrambling.Since the index of the scrambling pattern also has to be encoded in eachpacket, the number of useful bits would again be reduced to 8 or 9 perpacket. So for very short repeated messages, the un-scrambled shortpackets may be deemed to be most useful. For longer messages, guidedscrambling may be very useful.

Messages Constructed from Multiple Packets

For transmitting a useful amount of information from a light source to acamera receiver, messages are constructed which consist of aconcatenation of p packets, where each packet has its own bit content.Between each two packets, there is at least an inter-packet idle period(IPIP) to prevent ISI crosstalk between different packets. At the end ofa message, there is an extra inter-message idle period (IMIP). A messageconsisting of p packets is cyclically repeated.

In a preferred embodiment, p=3, so effectively 3 bytes of information(24 bits) are transmitted per message.

Inter-Packet Idle Period

The purpose of the inter-packet idle period (IPIP) is to limit the ISIinduced by the exposure time (Texp) of the camera to a single packet. Ina preferred embodiment, the duration of the IPIP shall be equal to themaximum expected exposure time, Texp_max. This may provide a universalIPIP format, since it allows recovery of the messages for any Texp if:Texp≤IPIP=Texp_max.

The inventors have also found that messages are recoverable ifTexp>IPIP, for carefully chosen TM-symbol rates, where the carefullychosen TM-symbol rates then depend on the actual Texp used by thecamera. Formats exploiting the enhanced signaling speed for this casewill belong to the “negotiated signal formats”, since the transmittinglight source and the camera receiver should agree on the choice oftransmit parameters such as TM-symbol rate, number of packets permessage, IPIP and/or IMIP, to ensure that the actual coded lighttransmissions can be received. The choice of these parameters depends onthe available camera settings of, e.g., Texp, frame rate, line rate andthe footprint of the light source.

Note, while embodiments herein are described in terms of an IPIPfollowing each packet and an extra IMIP being tagged on the end of thelast IPIP, in an alternative description or implementation an IPIP maybe included only between adjacent packets of the same message, with thetotal idle period following the end of the last message being the IMIP.

Inter-Message Idle Period

The inter-message idle period (IMIP) is an idle period that is appendedafter the last IPIP which trails the last packet of a message. The IMIPmay be measured in TM symbols.

The IMIP serves two goals:

(i) to make sure that the total message duration is such that itsatisfies nice “rolling properties” given the frame rate, i.e. such thatfootprints of consecutive frames reveal the complete message as fast aspossible; and/or

(ii) the second purpose of the IMIP is to provide an asymmetry in thepattern of packets and idle periods within the cyclic repetition ofmessages. This property can be used in the cyclic synchronization of areceiver.

Synchronization Elements of Format

For synchronization purposes, two elements of the signal format aresignificant.

(i) The usage of the first bit of each 9-bit packet as a synchronizationbit. In a preferred embodiment, the first bit of the first packet of amessage shall be one, while the first bit of all remaining packets shallbe zero.

(ii) The usage of the inter-message idle period (IMIP). The presence ofa non-zero IMIP breaks the regular temporal packet structure in arepeated message, because the total idle time after the last packet of amessage is longer than the idle times between the other packets.

In a preferred embodiment, the IMIP shall have a duration of at least 4symbols.

Example Parameters

Given all of the above considerations, some example parameter choicesare:

fsymbol ≥1 kHz (flicker and strobo),

envisioned packet durations:

-   -   around 52 ms (≥49 ms) for fsymbol ˜1 kHz    -   around 26 ms (≥24.5 ms) for fsymbol ˜2 kHz    -   around 13 ms (≥12.25 ms) for fsymbol ˜4 kHz,

message durations Tm are an integer multiple of packet durations, and/or

interesting message durations: around 26, 52, 104 ms.

For Instance:

the exposure time is less than or equal to ( 1/30)s, the symbol rate is1 kHz and the packet is 52 ms including inter-packet idle period;

the exposure time is less than or equal to ( 1/60)s, the symbol rate is2 kHz and the packet is 26 ms including inter-packet idle period; or

the exposure time is less than or equal to ( 1/120)s, the symbol rate is4 kHz and the packet is 13 ms including any inter-packet idle period.

Other Example Parameter Choices:

3-packets format (with CRC) having a duration of 158 ms @ 1 kHz symbolrate, with the 158 ms corresponding to a 3-byte message having an IPIPof 33 symbols and an IMIP of 2 symbols; or

a packet length of 70 symbols ˜35 ms @ 2 kHz, with the 35 mscorresponding to a 3-byte message having an IPIP of 3 symbols and anIMIP of 4 symbols (e.g. this format can be used where T_exp iscontrolled to be less than ( 1/500)s).

In a negotiated format case, the controller may be arranged to selectbetween a list of multiple combinations of parameters, comprising any ofone or more these combinations, and/or other combinations. In auniversal format, one particular combination is pre-chosen to satisfy asmany cameras (or rather exposure times) as possible.

Cyclic Redundancy Check (CRC)

In a preferred embodiment, a message consists of several packets, whereeach packet contains 1 byte of information. In case a CRC is used, it issuggested that the last byte of each message is an 8-bit CRC. Because ofthe repeated decoding results delivered by a receiver decoding thecyclically repeated signal format, one can obtain potentially manyrealizations of the transmitted message, which allow the reliability ofa received message to be enhanced by comparing the decoding results ofconsecutive decoded variants of the same message.

In a preferred embodiment, the CRC is characterized by a pre-load andparity inversion. The pre-load can be application-specific, thusallowing a receiver to distinguish between messages from differentapplications in use in the same environment. Note that there is atrade-off between the number of different pre-loads in use, and theeffective error-detection capability of the CRC

Multiple Messages

The inventors have found that one can transmit a concatenation ofdifferent messages where each message m_(i) is repeated N times, where Nis a sufficient number of times such that a camera receiver canreconstruct reliably a complete message m_(i) given the footprint of thetransmitting light source. After N repetitions of the same message thelight source can transmit a completely different message m_(i+1) havingthe same signal parameters, by just concatenating, say N, repetitions ofmessage m_(i+1).right after m_(i). It turns out that a receiver iscapable of recognizing a coherently reconstructed message by observingthe CRC.

Message Reassembly

The following describes a process of reassembling or “stitching” ofvideo frames for coded light message recovery by a camera. The receiverreceives a signal formatted as described above and re-assembles theparts of the message into a complete message, which is then provided forfurther processing.

In embodiments the reassembly process comprises the following.

(i) For each of multiple frames, establish a sample per image line asdescribed above (see again the samples 19 taken from lines 18 in FIG.2).

(ii) Collect all the (active) samples of a given frame into atime-sequence (each positioned at the respective time at which thesample from that line was located within the frame). This sequence formsa “marginal signal” or “frame signal” for each frame.

(iii) Next extend the signals with zeros resulting in an “extendedmarginal signal” or “extended frame signal”, where the duration of eachextended signal is n times the message duration (n being an integer) andwhere the duration is longer than the frame duration.(iv) Next the active samples are time-aligned, i.e. shift the samplesper line by Tframe to the right within the time frame or scale definedby the extended signal. This is done cyclically, i.e. in a wrap-aroundfashion wrapping around beyond the end of the extended frame signallength. This way the shifted position of the samples within the extendedframework is such that it facilitates reassembly.(v) Next the samples are collapsed (i.e. reassembled). In embodimentsdifferent reconstructions can be found by shifting one measurementfurther.

Once reconstructed the signal can be filtered to eliminate inter-symbolinterference (ISI), e.g. using a Wiener filter.

In embodiments, the ISI filter is robust enough to handle gaps in thereassembled data (this robustness being at least in part a result of themodulation code, message format and the Wiener filter). The process mayalso allow elegant handling of skipped frames.

In further embodiments, as an additional feature, the process may alsoallow the receiver to correct for clock deviations relative to thetiming of Tm or Tframe based on correlation of reconstructed signals.

An example of the message reassembly process will be discussed in moredetail shortly, but first some example details of the receiver front endare elaborated upon with reference to FIGS. 1 to 4.

In embodiments, the camera-based, digitally-coded light receiverdisclosed herein is very different from the class of well-knownreceivers of digital signals using radio or IR communication. Both thegeneral structure of the coded light receiver, as well as the detailedalgorithms for performing the sub-tasks within a coded light receiver,are quite distinct.

The input of a camera-based coded light receiver consists of a movietaken in a known format. For instance, a well-known video format is480p, a progressive scan format having frames taken at 29.97 frames persecond (fps), where each frame consists of 480 lines and each linecontains 640 pixels. The coded light receiver consists of the digitalsignal processing applied to this movie for obtaining the digitalcontent of the modulated light source.

The signal processing performed by the receiver may comprise 2D signalprocessing and 1D signal processing. The 2D signal processing maycomprise:

(i) selection of an appropriate color (R, G or B) or a linearcombination of colors for extracting the coded light signal;

(ii) image segmentation using a blob approach, efficiently identifyingregions in the image containing coded light sources;

(iii) identifying spatial filter “active pixels” within each blob;

(iv) efficient motion compensation (independently for each source) usingmarginal; and/or

(iv) computing signal “marginal” by combining of active pixels per line(computing samples 19 resulting from each line 18 in FIG. 2).

The 1D signal processing may comprise:

(i) using correlations within a frame for estimating the transmit clock(works best for footprint>>duration of message);

(ii) assuming the use of the above-described signal format, where amessage is cyclically repeated by the transmitter, and exploiting theknowledge of the repetition time of the message (Tm) and the knowledgeof the number of frames per second (Tframe) for reconstructing acomplete message from the partial snapshots obtained in each frame (thisis the reassembly process to be described in more detail shortly);(iii) using correlations between successive reconstructed signals forestimating the transmit clock;(iv) using robust Wiener filtering on a single period of the message formitigating the ISI caused by Texp;(v) applying robust Wiener interpolation if the reassembly procedure hasleft holes in the reconstruction;(vi) finding global circular synchronization by processing using a synctemplate;(vii) decoding the bits by making decisions on the optimum samplingpoints given by the global circular synchronization; and/or(viii) checking CRC on consecutive reconstructed messages. If m out of nconsecutive reconstructions have CRC=OK, accept the message.

For a particular message format and a given footprint, it may take forexample 30 consecutive frames for reassembling a complete message. Ifone has a recording of 2 seconds (say, 60 frames), the receiver cangenerate 31 different realizations of the same message. In embodiments,by comparing theses different decoding results it is possible to aidsynchronization of the receiver clock with the received signal.

Regarding the selection of appropriate color, it turns out that thesection of the appropriate color can be significant for recovering thecoded light signal. For instance, the color Green (G) is characterizedby the highest pixel density in the camera, thus giving the highestspatial (and thus temporal) resolution of a coded light signal. This maybe of importance if the coded light is using a high symbol frequency(wide bandwidth). On the other hand, it turns out that the color blue(B) is favorable if a light source has a high intensity and if Texp israther long, since this color tends to lead to less clipping of thepixels.

Referring to FIGS. 2 to 4, for image segmentation, embodiments of thepresent disclosure use a “blob”-approach for recognizing regions in animage that can be associated with a lamp possibly transmitting codedlight information. Typically, a blob is a region of high intensity in animage (e.g. see FIG. 3). An algorithm can recognize and differentiatedifferent blobs in an image. For example using edges of the blobs allowsfor efficiently tracking a blob and limiting the 2D signal processingassociated with each blob in the different frames of the video sequence.

To find contributing pixels within a blob, only those pixels that aremodulated, i.e., have sufficient intensity variations due to themodulated light source, contribute effectively to the signal. Othersource pixels effectively only produce “noise” or other unwanted sideeffects. Typically, pixels that are clipped are also removed fromfurther consideration (e.g. see FIG. 4). Also pixels having insufficientintensity are removed. The resulting set of “contributing pixels”belonging to a light source can be represented as a binary spatial 2Dfilter.

The following describes of an algorithm that operates on the samplesthat are obtained as the “marginals” in each frame (the samples 19 inFIG. 2, i.e. the “line-combined” samples).

FIG. 3 depicts a receiver-generated binary picture that indicates thesource of interest. FIG. 4 shows, in binary, the contributing pixels ofthe selected source in each frame. Note that the pixels in the centerpart of the source do not contribute, because those pixels areover-exposed, i.e. clipped.

FIG. 13 shows the “marginal signals” (made up of the samples 19 fromFIG. 2) of each of the 100 consecutive frames shot in the movie, eachsample obtained by a proper operation of the active pixels of itscorresponding line. I.e. each “marginal signal” is the signal obtainedfrom one given frame 16, with the samples 19 taken from each active line18 of that frame being positioned in time at the respective times withinthe frame duration at which they were sampled. Note that in FIG. 2 thetime axis corresponding to consecutive samples of a single frame runsfrom top to bottom, while in FIG. 13, the time axis of consecutivesamples in a single frame runs from left to right (with the page inlandscape). In FIG. 13, the 100 consecutive video frames (each framegenerating a single 1-dimensional signal) are stacked on top of eachother, consecutive frames going from top to bottom.

Note also that, although Tframe equals about 1/30˜33 ms, the marginalsignal of a single frame has a duration of only about 26.5 ms due to thehidden lines 26. At the bottom of FIG. 13 is shown a bar 46 thatindicates the samples that originate from the lines that cover the lightsource in each frame, i.e. only this part of each row contains samplesthat originate from the source. For this example, it turns out that thefootprint of the source with respect to a frame, FSF˜0.14, i.e. onlyabout 14% of the lines per frame actually contain pixels of the source.

In FIG. 14 it is shown how to use the known durations Tm and Tframe togenerate “extended marginal signals” or “extended frame signals”, eachbeing an extended version of the signal sampled from a respective frame.This is done as follows.

(i) Define for each frame a stretch, i.e. a temporal region around (e.g.extending after) the active samples of FIG. 13, such that a stretchduration of m times Tm is obtained, where m is a conveniently choseninteger. Note that zeros can always be added or removed outside theactive samples.(ii) Compute num_periods=ceiling(Tframe/(m*Tm)), where “ceiling” meansround up to the nearest integer.(iii) Cyclically repeat each stretch num_periods times such that an“extended marginal signal” is obtained for each frame having a totalduration of at least Tframe. Note that the extended marginal signalalways has a duration that is larger than Tframe, and that it is aninteger multiple of Tm.

In the example, Tm=158 ms; Tframe=33.36, so m=1 and num_periods=1, andeach frame is extended by zeros to obtain a stretch of 158 ms (=1 periodof the message). Note that the actual useful observation in each frame(stretch) is only a fraction of about 0.03 of a complete message,indicated by the bar 48 in FIG. 14. One may say that the footprint ofthe source with respect to a message, FSM, ˜0.03.

Note that in embodiments it is not necessary to use two separateintegers m and num_periods. The point is to determine a time period thatis an integer multiple of the message length (duration) Tm, and which islonger than the frame length (duration) Tframe. This period defines areference time scale or reference frame within which the signalsobtained from the different frames can be aligned, as now disused.

The time-alignment of the observations originating from the differentframes is performed using Tframe and the now-defined reference frameworkor scale determined as explained above. The “extended marginal signal”of each line is shifted Tframe to the right (in the positive timedirection) with respect the extended marginal signal of its previousframe. However, as the extended marginal signals were made a multiple ofthe message duration Tm, and because the transmitted message is repeatedcyclically, one can replace the shift of each extended marginal signalby a cyclic (wrap around) shift, thus obtaining the results in FIG. 15.

That is, as mentioned, the “extending” discussed above provides a timingreference period, which defines scale or framework within which toposition the signals obtained from each frame. This reference period hasa length being an integer multiple of the message duration Tm.Furthermore, the scale or framework it defines wraps around. I.e. beyondthe end of the timing reference period, the scale or framework wrapsback around to the beginning of the reference period. Hence if inshifting the signal from a given frame right by Tframe relative to itspreceding frame causes a portion of that frame's signal to shift “offthe end” or “off the right hand side” of the reference scale or frame(beyond the timing reference period, i.e. beyond the integer multiple ofTm that has been defined for this purpose), then the portion of thatsignal continues by reappearing at the beginning of the reference scaleor frame (starting from time zero relative to the timing referenceperiod).

Note that in embodiments, it need not be necessary to “extend” thesignals from each frame (the “marginal signals”) by adding zeros. Thisis just one way of implementing the idea of creating a wrap-aroundreference frame that is an integer multiple of the message duration Tm.An equivalent way to implement or consider this would be that thistiming reference period (that is an integer multiple of Tm) defines a“canvas” on which to place the signals from each frame, and on which toshift them by their respective multiples of Tframe in a wrap-aroundmanner.

Note also that in all cyclically-shifted, extended marginal signals, thereceiver keeps track of the locations of the active samples originatingfrom the coded light source.

Having results as in FIG. 15, the receiver can now, for each time sampleof the message, look in the vertical direction for frames that have avalid contributing sample at that position.

From the FSM being about 0.03, one can expect that it takes at least(0.03)−1˜33 frames for recovering a complete message. Typically, becauseof overlap, in embodiments the receiver may need about twice that manyframes for complete recovery.

From FIG. 16, it can be seen that the decoder, in the example, needs 70consecutive frames for a reconstruction of a complete message (a movieof ˜2 seconds). Since each 70 consecutive frames give a reconstruction,a video of 100 frames gives 31 different reconstructions (albeit theyare dependent).

FIG. 17 shows the result of the reconstruction 48 (and output of therobust Wiener equalization 50) of the first reconstructed message. Thelength of the bold bar 52 at the top of the figure indicates whichfraction (˜3%, ˜5 ms) of the complete message is obtained from a singleframe in this example.

In further embodiments, the procedure described above also can deal withso-called “skipped frames”. The assumption is that a possibly skippedframe is detected by observing the frame recording times that are givenby the camera. If a frame is skipped, the corresponding marginal signalwill obtain no valid support in FIGS. 16 and 17. Therefore thereassembly algorithm will automatically discard the corresponding frame.

In yet further embodiments, by observing the correlations betweendifferent reconstructed signals (31 of them in FIG. 17), one can correctfor clock deviations between transmitter and receiver. If all clocks arein perfect synchronism (assuming perfect knowledge of Tm and Tframe),these different reconstructed signals will be perfectly alignedvertically (modulo different noise effects). A clock deviation will showup as a non-zero shift of the best correlation. In this manner, thereceiver can adapt to the transmitter clock. It turns out that thismethod works, even if the received signal is heavily corrupted by theISI caused by the exposure time of the camera.

The minimum number of frames required in order to get a completereassembly is now discussed.

Consider again what happens to covering a message with footprints if:

relative footprint α=Tfootprint/Tf=0.4

0<α≤1, (in practice e.g. 0<α≤0.88 due to hidden lines)

If Tm is about Tframe, the alignment of the messages looks like FIG. 18.

If Tm is about 1.5 times Tframe, the alignment of the messages lookslike FIG. 19.

It turns out that, if α<1, one obtains “non-rolling” footprints if themessage durations Tm are a multiple of the frame duration Tf. If α<0.5,one obtains “switching” footprints if Tm is a half-integer multiple ofTf (0.5, 1.5, 2.5, . . . ).

As discussed previously in relation to FIG. 8, in general, if1/(n+1)<α≤1/n, where n is integer, then one has non-rolling messagedurations Tm if:

$\left. {{{\frac{T_{m}}{T_{f}}\epsilon\left\{ \frac{k}{m} \right.m} = 1},\ldots\mspace{14mu},n,{k\;\epsilon\; N^{+}}} \right\}$

Note that the singularities for small m are wider than for larger m.

For a non-rolling message duration Tm=T₀, define m₀, the smallest m suchthat m₀·T₀=k₀·Tframe, as the order of the non-rolling T₀. GCD(m₀,k₀)=1.

The numbers m₀ and k₀ determine the repeat pattern of footprints andmessages in the neighborhood of T₀: about k₀ non-rolling footprints gointo m₀ messages.

Consider a message of duration Tm˜T₀ in the neighborhood of anon-rolling message duration T₀: after 1 round of m₀ messages, there arek₀ disjoint equidistant footprints partly covering the message.

The non-covered part is: T₀−k₀·α·Tframe, divided into k₀ equal parts ofsize Tg, whereTg=(T ₀ −k ₀ ·Tframe·α)/k ₀=(T ₀ −m ₀ ·T ₀·α)/k ₀ =T ₀(1−m ₀·α)/k ₀

FIG. 18 shows the time-alignment of messages with consecutive footprintswhere α=0.4; close to m₀=1, k₀=1. Here the message does not rollsignificantly and each frame sees almost the same part of the message(rolling only very slowly).

FIG. 19 shows the time-alignment of messages with consecutive footprintsin another case where α=0.4; close to m₀=2, k₀=3. Here one sees“switching”.

After 1 round of m₀ messages, there are k₀ gaps each of duration Tg thathave to be covered by the incremental shifts of the footprints in thenext rounds.

Considering the shift ΔT of footprints from one to the next round:ΔT=m0·|Tm−T0| [ms]

one needs ˜1+Tg/ΔT rounds to cover the complete message

1+Tg/ΔT rounds correspond to Nf=(1+Tg/ΔT)·k₀ frames

$\begin{matrix}{N_{f} \approx {\left( {1 + \frac{T_{g}}{\Delta\; T}} \right) \cdot K_{0}}} \\{= {k_{0} + \frac{T_{0}\left( {1 - {m_{0} \cdot \alpha}} \right)}{{{m_{0}T_{m}} - {m_{0}T_{0}}}}}} \\{{= {k_{0} + \frac{T_{0}\left( {1 - {m_{0} \cdot \alpha}} \right)}{m_{0}{{T_{m} - T_{0}}}}}},{m_{o} \leq n}}\end{matrix}$

Note the hyperbolic behavior of Nf for Tm in neighborhood of T₀. Notealso the effect of m₀ and T₀ on the “width” of a singularity.

Robust Wiener Filtering

This following describes another part of the decoder which inembodiments allows the above implementation to have a considerablybetter performance and allows the device to be used with a much widerrange of cameras.

A robust Wiener filter is introduced, that can be used, e.g. forequalizing a signal that is corrupted by a filter H(f) having unknownparameters, and by additive noise. The robust Wiener is a constantfilter that produces optimum results in an MSE sense, assuming that theprobability distribution of the filter parameters is known.

Wiener filter theory in itself is well-known in digital signalprocessing, and has been used extensively since the second world war.Wiener filters can, for instance, be used for estimation of a (linearly)distorted signal in the presence of noise. A Wiener filter (equalizer)then gives the best (mean square error, MSE) result.

In classical (frequency-domain) Wiener filtering, e.g. de-convolution,one has two independent, stationary, zero mean random processes X and N₀as shown in FIG. 20.

In a typical application, X represents an input signal input to a filterH (numeral 54 in FIG. 20), and N₀ represents additive noise added at theoutput of the filter H. The Wiener filter G (numeral 56) is arranged toequalize the filter H, i.e. to undo the effect of the filter H on theinput signal X in presence of the noise N (to a best approximation).

A typical application is the detection of coded light with a rollingshutter camera. In this case, the equivalent digital signal processingproblem corresponds to the restoration of a digital signal that has beenfiltered by a temporal box function. See FIG. 21. That is, the inputsignal X represents the coded light signal as captured by the rollingshutter camera, and the filter H represents the filtering effect of therolling shutter acquisition process. This filter H is created by theexposure of each line. It amounts to a box function (i.e. rectangularfunction) in the time domain with a width Texp—i.e. a line is exposedfor a time Texp in which time it captures the signal (the transferfunction of the filter H in the time domain is uniformly “on”), andbefore and after that it does not capture any signal (the transferfunction of H in the time domain is zero). A box function in the timedomain corresponds to a sinc function in the frequency domain. An effectof this filter can be to produce inter-symbol interference. Hence in thefollowing, the filter created by Texp may be referred to in terms of itsunwanted effect, as an “ISI filter”.

(FIGS. 21 and 22 also show how the noise N₀ may be considered as the sumof: (i) a noise term n1 at the input of the filter H passed through thefilter H, and (i) a noise term n2 at the output of the filter H.)

The task is to find a linear filter G which provides a minimum meansquare error estimate of X using only Y. To do this the Wiener filter Gis preconfigured based on assumed knowledge of the filter H to beequalized (i.e. undone), as well as N₀. It is configured analyticallysuch that (in theory given knowledge of H and the spectrum of X and N),applying the Wiener filter G to Y (where Y is the input signal X plusthe noise N) will result in an output signal X^ that minimizes the meansquare error (MSE) with respect to the original input signal X.

The classical Wiener filter formulation (in the frequency domain) is:

${G(f)} = \frac{{H^{*}(f)}{S(f)}}{{{{H(f)}}^{2}{S(f)}} + {N(f)}}$where S(f) is the spectral density of the input signal X and N(f) is thespectral density of the noise term N₀.

As can be seen, the formulation of a Wiener filter comprises arepresentation of the filter to be equalized, in this case in the formof H* and |H|² (=HH*). Traditionally in the classical Wiener filter, itis assumed that H(f), the filter to be equalized, and N(f), the noisespectral density, are exactly known. In the case of equalizing for theISI filter created by a rolling shutter acquisition process, thisimplies exact knowledge of Texp. It is also assumed that the spectraldensities S(f) and N₀(f) of the processes X and N, respectively, areknown.

However, Wiener filters are in fact very sensitive to errors in theestimation of H(f). Some techniques have been developed in the past todeal with an unknown distortion, such as

iterative (time-consuming) approaches, where one tries to vary thetarget response until one gets the best result; or

min-max approaches, where one tries to identify the worst case H(f) andoptimizes the Wiener filter for this.

A problem therefore in using classical Wiener filtering forequalization, is in applying this theory if the gain of the filter hasto be large and the filter to be equalized is not known very accurately.

E.g. for a bandwidth of the signal is in the order of 1 kHz with Texp inthe range of 1/30 of a second, the ISI filter can introduce severeinter-symbol interference (ISI) like shown in FIGS. 11 and 12.

In order to undo this ISI at the receiver side, it would be desirable toprovide a “powerful” equalizer filter that is insensitive toinaccuracies in the definition of H(f).

According to the present disclosure, this can be achieved by computing afixed “average Wiener filter”, a Wiener-like filter that is robust underunknown variations of the ISI filter H(f). This “robust Wiener filter”can produces a more optimal output in terms of MSE, given a statisticaldistribution of the relevant parameters of H(f).

In an application to coded light, this theory allows one to reconstructa coded light signal where Texp of the camera is only knownapproximately, which can often be the case.

The inventors have found a particularly efficient derivation of anoptimal robust Wiener filter. In the following the problem is describedin the frequency domain (so in terms of H(f), as introduced before).Note that in an application to coded light, the robust Wiener filter maybe constructed in real time in a camera-based (smart phone) decodingalgorithm, as Texp, and therefore H(f), is defined or changed during theactual read-out of a lamp.

The robust Wiener filtering is based on noting that H(f) is not knownexactly, but may in fact be dependent on at least one unknown quantityθ, i.e. a parameter of H whose value is not known and may in fact in anygiven case be found within a range of values, e.g. between two limits −Δand +Δ (or more generally Δ1 and Δ2). That is, it is assumed that thefilter H(f;θ) depends on a random parameter θ, independent of X and N.

For a box function of width θ, i.e. a sinc in the frequency domain, onemay write:

${H\left( {f;\theta} \right)} = \frac{\sin\left( {{\pi\theta}\; f} \right)}{{\pi\theta}\; f}$

And in the case of an ISI filter created by the box, θ is Texp.

The robust Wiener filter 56′ is then created by taking the classicalWiener filter representation given above, and where a representation ofthe filter to be equalized appears, replacing with a correspondingaveraged representation that is averaged over the potential values ofthe unknown parameter θ (e.g. average between −Δ and +Δ or moregenerally Δ1 and Δ2). That is, wherever a term based on H(f) appears,this is replaced with an equivalent averaged term averaged with respectto θ.

Starting from the classical formulation above, this gives:

$G = \frac{{E_{\theta}\left\lbrack H^{*} \right\rbrack} \cdot S}{{{E_{\theta}\left\lbrack {HH}^{*} \right\rbrack} \cdot S} + \mathcal{N}_{0}}$where E is the average with respect to θ. See also FIG. 23.

A derivation of this is now explained in further detail. It is desiredto find a fixed linear filter G that provides a linear minimum meansquare error estimate{circumflex over (X)}(f)=G(f)Y(f)such thate(f)=E _(X,N,θ)[(X(f)−{circumflex over (X)}(f))²]is minimal.

Extending the classical derivation by taking also the ensemble averagewith respect to θ, one obtains:

$\begin{matrix}{e = {E_{X,N,\theta}\left\lbrack {{X - \hat{X}}}^{2} \right\rbrack}} \\{= {E_{X,N,\theta}\left\lbrack {{X - {G\left( {{HX} + N} \right)}}}^{2} \right\rbrack}} \\{= {E_{X,N,\theta}\left\lbrack {{{\left( {1 - {GH}} \right)X} - {GN}}}^{2} \right\rbrack}} \\{{= {{{E_{\theta}\left\lbrack {\left( {1 - {GH}} \right)\left( {1 - {GH}} \right)^{*}} \right\rbrack} \cdot {E\left\lbrack {X}^{2} \right\rbrack}} + {{GG}^{*} \cdot {E\left\lbrack {N}^{2} \right\rbrack}}}},} \\{{{since}\mspace{14mu} X},{{N\mspace{14mu}{and}\mspace{14mu}\theta\mspace{14mu}{are}\mspace{14mu}{independent}\mspace{14mu}{and}\mspace{14mu}{E\lbrack X\rbrack}} =}} \\{= {{{E_{\theta}\left\lbrack {\left( {1 - {GH}} \right)\left( {1 - {GH}} \right)^{*}} \right\rbrack} \cdot S} + {{GG}^{*} \cdot \mathcal{N}_{0}}}} \\{= {{\left\{ {1 - {G \cdot {E_{\theta}\lbrack H\rbrack}} - {G^{*} \cdot {E_{\theta}\left\lbrack H^{*} \right\rbrack}} + {{GG}^{*} \cdot {E_{\theta}\left\lbrack {HH}^{*} \right\rbrack}}} \right\} \cdot S} + {{GG}^{*} \cdot \mathcal{N}_{0}}}}\end{matrix}$The best G(f) is found by differentiating e to G and setting the resultto 0:

${\frac{\partial}{\partial G}e} = {{{\left\{ {{- {E_{\theta}\lbrack H\rbrack}} + {G^{*} \cdot {E_{\theta}\left\lbrack {HH}^{*} \right\rbrack}}} \right\} \cdot S} + {G^{*} \cdot \mathcal{N}_{0}}} = 0}$from which one obtains:

$G = \frac{{E_{\theta}\left\lbrack H^{*} \right\rbrack} \cdot S}{{{E_{\theta}\left\lbrack {HH}^{*} \right\rbrack} \cdot S} + \mathcal{N}_{0}}$In a similar manner, one can incorporate a target response of a matchedfilter (MF):

$G = \frac{H_{MF} \cdot {E_{\theta}\left\lbrack H^{*} \right\rbrack} \cdot S}{{{E_{\theta}\left\lbrack {HH}^{*} \right\rbrack} \cdot S} + \mathcal{N}_{0}}$

To apply this there remains the computation of E_(θ)[H] and E_(θ)[HH*].Some examples are given below.

A first approach is to use a Taylor series expansion of H and moments ofθ. In the coded light rolling shutter application θ=Texp.

E[θ] = θ̂ E[(θ − θ̂)²] = σ_(θ)²${H^{\prime}\left( {f;\hat{\theta}} \right)} = {\frac{\partial}{\partial\theta}{H\left( {f,\theta} \right)}{_{\theta = \hat{\theta}}{{H^{''}\left( {f;\hat{\theta}} \right)} = {\frac{\partial^{2}}{\left( {\partial\theta} \right)^{2}}{H\left( {f,\theta} \right)}}}}_{\theta = \hat{\theta}}}$A Taylor series expansion gives:

$\mspace{20mu}{{H\left( {f;\theta} \right)} = {{H\left( {f;\hat{\theta}} \right)} + {\left( {\theta - \hat{\theta}} \right){H^{\prime}\left( {f;\hat{\theta}} \right)}} + {\left( {\theta - \hat{\theta}} \right)^{2}\frac{H^{''}\left( {f;\hat{\theta}} \right)}{2}} + {O\left( {\theta - \hat{\theta}} \right)}^{3}}}$$\mspace{20mu}\begin{matrix}{{E_{\theta}\left\lbrack {H\left( {f;\theta} \right)} \right\rbrack} = {{H\left( {f;\hat{\theta}} \right)} + {{H^{\prime}\left( {f;\hat{\theta}} \right)}{E_{\theta}\left\lbrack {\theta - \hat{\theta}} \right\rbrack}} +}} \\{{\frac{H^{''}\left( {f;\hat{\theta}} \right)}{2}{E_{\theta}\left\lbrack \left( {\theta - \hat{\theta}} \right)^{2} \right\rbrack}} + {O\left( {\theta - \hat{\theta}} \right)}^{3}} \\{= {{H\left( {f;\hat{\theta}} \right)} + {\frac{H^{''}\left( {f;\hat{\theta}} \right)}{2}\sigma_{\theta}^{2\;}} + {O\left( {\theta - \hat{\theta}} \right)^{3}}}}\end{matrix}$E_(θ)[H(f; θ)H^(*)(f; θ)] = H(f; θ̂)H^(*)(f; θ̂) + (H^(′)(f; θ̂)H^(′*)(f; θ̂) + Re{H(f; θ̂)H^(″*)(f; θ̂)}) ⋅ σ_(θ)² + O(θ − θ̂)³In the rolling shutter application:

${H\left( {f;\theta} \right)} = \frac{\sin\left( {{\pi\theta}\; f} \right)}{{\pi\theta}\; f}$Then:

${\frac{\partial}{\partial\theta}{H\left( {f;\theta} \right)}} = {\frac{1}{\theta}\left\{ {{\cos\left( {{\pi\theta}\; f} \right)} - \frac{\sin\left( {{\pi\theta}\; f} \right)}{{\pi\theta}\; f}} \right\}}$${\frac{\partial^{2}}{\left( {\partial\theta} \right)^{2}}{H\left( {f;\theta} \right)}} = {{\frac{2}{\theta^{2}}\left\{ {\frac{\sin\left( {{\pi\theta}\; f} \right)}{{\pi\theta}\; f} - {\cos\left( {{\pi\theta}\; f} \right)}} \right\}} - {\frac{\pi\; f}{\theta}{\sin\left( {{\pi\theta}\; f} \right)}}}$

This approach works better for low frequencies since H″(f,θ) blows upwith increasing frequency.

A second approach is to use a more exact computation assuming a knowndistribution of θ. Example: θ is uniform distributed between θ^−Δ andθ^+Δ, and

${H\left( {f;\theta} \right)} = {\frac{\sin\left( {{\pi\theta}\; f} \right)}{{\pi\theta}\; f}.}$Then:

$\begin{matrix}{{E_{\theta}\left\lbrack {H\left( {f;\theta} \right)} \right\rbrack} = {\int_{\hat{\theta} - \Delta}^{\hat{\theta} + \Delta}{\frac{\sin\left( {\pi\; f\;\theta} \right)}{\pi\; f\;\theta}{\mathbb{d}\theta}}}} \\{\approx {\frac{- 1}{\hat{\theta}}\frac{1}{\left( {\pi\; f} \right)^{2}}\left\{ {{\cos\;\pi\;{f\left( {\hat{\theta} + \Delta} \right)}} -} \right.}} \\{{\left. {\cos\;\pi\;{f\left( {\hat{\theta} - \Delta} \right)}} \right\}\frac{1}{2\Delta}},{\Delta{\operatorname{<<}\hat{\theta.}}}}\end{matrix}$ $\begin{matrix}{{E_{\theta}\left\lbrack {{H\left( {f;\theta} \right)}{H^{*}\left( {f;\theta} \right)}} \right\rbrack} = {\int_{\hat{\theta} - \Delta}^{\hat{\theta} + \Delta}{\frac{\sin^{2}\left( {\pi\; f\;\theta} \right)}{\left( {\pi\; f\;\theta} \right)^{2}}{\mathbb{d}\theta}}}} \\{{{\approx {\frac{1}{4{\Delta\left( {\pi\; f\;\hat{\theta}} \right)}^{2}} \cdot \left\{ {{2\Delta} - \frac{\sin\left( {2\pi\; f\;\theta} \right)}{2\pi\; f}} \right\}}}}_{\hat{\theta} - \Delta}^{\hat{\theta} + \Delta},{\Delta{{\operatorname{<<}\hat{\theta}}.}}}\end{matrix}$

Although in embodiments the above has been described in terms of acertain modification to the classical Wiener frequency domainformulation, there may be other Wiener filter formulations (e.g. timedomain or approximations of a Wiener filter, or formulations solved fora particular H) and the principle of replacing an assumed-to-be-known Hor function of with an average H or function of H may also be applied insuch formulations.

Note also the robust Wiener filter disclosed herein can be used toequalize other filters other than a box (rectangular) filter, and/or inother applications other than receiving coded light. Another example isa band pass filter having a center frequency f₀ which may not be exactlyknown. In this case the filter to be equalized is a function offrequency f and center frequency f₀, H(f; f₀), and the robust Wienerfilter is determined from an averaged representation of H(f; f₀)averaged with respect to f₀. E.g.:

$G = \frac{{E_{f\; 0}\left\lbrack H^{*} \right\rbrack} \cdot S}{{{E_{f\; 0}\left\lbrack {HH}^{*} \right\rbrack} \cdot S} + \mathcal{N}_{0}}$

Further, the idea of the robust Wiener filter can also be extended to ahigher dimensional theta, i.e. more than one parameter may be allowed tobe uncertain. In this case the representation of filter H to beequalized (e.g. H* and HH*) is averaged over each of the unknownquantities. For example, the parameters may be the center frequencyand/or band width of a band pass filter.

Further, the noise term N₀ could alternatively or additionally representthe spectral density of an interfering signal. A generic term for noiseand/or interference is “disturbance”.

It will be appreciated that the above embodiments have been describedonly by way of example. Other variations to the disclosed embodimentscan be understood and effected by those skilled in the art in practicingthe claimed invention, from a study of the drawings, the disclosure, andthe appended claims. In the claims, the word “comprising” does notexclude other elements or steps, and the indefinite article “a” or “an”does not exclude a plurality. A single processor or other unit mayfulfill the functions of several items recited in the claims. The merefact that certain measures are recited in mutually different dependentclaims does not indicate that a combination of these measured cannot beused to advantage. A computer program may be stored and/or distributedon a suitable medium, such as an optical storage medium or a solid-statemedium supplied together with or as part of other hardware, but may alsobe distributed in other forms, such as via the Internet or other wiredor wireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

The invention claimed is:
 1. A device comprising: an input for receivinga signal from a rolling-shutter camera which captures frames of a givenduration at a given frame rate by exposing a plurality of lines of eachframe in sequence, the signal comprising a coded light signal; a signalprocessing module connected to said input and arranged to obtain, viasaid input, a respective sample of the coded light signal from each of anumber of said lines, said number of lines being exposed over aduration, and said number being a substantially smaller number of saidlines than the plurality of lines exposed by the camera in each frame;wherein the coded light signal comprises multiple repetitions of amessage wherein the message has a duration longer than the duration overwhich said number of lines are exposed, and wherein said multiplerepetitions of the message are received with a timing such that adifferent part of the message is seen by the camera in each of aplurality of different ones of said frames; wherein the signalprocessing module, based on the frame duration and message duration, isconfigured to time align the different parts of the message from theplurality of different frames; and reassemble the message from thetime-aligned parts by performing at least the following steps: (i) foreach of multiple frames, establish a sample per image line; (ii) collectall active samples of a given frame into a time-sequence, eachpositioned at the respective time at which the sample from that line waslocated within the frame, thereby forming a frame signal for each frame;(iii) time-aligning the frame signals across the message duration; (iv)reassembling the message from the time-aligned frame signals.
 2. Thedevice of claim 1, wherein the whole message is seen over said pluralityof frames.
 3. The device of claim 1, wherein the message has a durationlonger than one frame.
 4. The device of claim 1, wherein the signalprocessing module is configured to perform said time-alignment by:determining a timing reference period that is an integer multiple of thelength of said message; and offsetting the part of the message receivedin each successive frame by the frame length with respect to itspreceding frame, but wrapping around to the beginning of said timingreference period beyond the end of said timing reference period.
 5. Thedevice of claim 4, wherein each frame comprises active lines andinactive lines, and said plurality of lines being the active lines, andthe signal processing module is configured to obtain a respective samplefrom each of the plurality of active lines of each frame including saidnumber of lines, thereby producing a frame signal having said frameduration; and the offsetting is performed by extending each frame signalto have a duration of the timing reference period.
 6. The device ofclaim 5, wherein said extending is performed by adding zeros.
 7. Thedevice of claim 1, wherein the signal processing module is configured todiscard one or more skipped frames.
 8. The device of claim 1, whereinthe signal processing module is configured to generate a plurality ofreassembled versions of the message each based on a different respectivesubset of said message parts, and to perform a synchronization between aclock of said device and a clocking of said coded light signal based ona correlation between said versions of the message.
 9. The device ofclaim 1, wherein each of said samples is taken by combining pixel valuesof the respective line.
 10. The device of claim 1, wherein said numberof lines is less than or equal to 14% of the lines of each frame. 11.The device of claim 1, wherein each of said parts is less than or equalto 3% of the message.
 12. The device of claim 1, wherein the number oflines from which said samples are obtained excludes lines comprising oneor more pixels that are over exposed.
 13. A receiver comprising thedevice of claim 1, and the camera.
 14. A system comprising the receiverof claim 13 wherein the coded light signal is transmitted from a lightsource; wherein the camera is positioned at a distance from the lightsource such that said samples are obtained from the substantiallysmaller number of lines than are exposed by the camera in each frame andthe message is longer than said number of lines.
 15. A methodcomprising: receiving a signal from a rolling-shutter camera whichcaptures frames by exposing a plurality of lines of each frame insequence, the signal comprising a coded light signal; and obtaining arespective sample of the coded light signal from each of a number ofsaid lines, said number of lines being exposed over a duration, and saidnumber being a substantially smaller number of said lines than theplurality of lines exposed by the camera in each frame; wherein thecoded light signal comprises multiple repetitions of a message whereinthe message has a duration longer than the duration over which saidnumber of lines is exposed, and wherein the message is repeated multipletimes with a timing such that a different part of the message is seen bythe camera in each of a plurality of different ones of said frames;wherein the method further comprises, based on the frame duration andmessage duration, time aligning the different parts of the message fromthe plurality of different frames, and reassembling the message from thetime-aligned parts by performing at least the following steps: (i) foreach of multiple frames, establish a sample per image line; (ii) collectall active samples of a given frame into a time-sequence, eachpositioned at the respective time at which the sample from that line waslocated within the frame, thereby forming a frame signal for each frame;(iii) time-aligning the frame signals across the message duration; (iv)reassembling the message from the time-aligned frame signals.
 16. Acomputer program product comprising a plurality of program codeportions, stored in a non-transitory computer readable medium, andconfigured so as when executed on a processor perform operations of:receiving a signal from a rolling-shutter camera which captures framesby exposing a plurality of lines of each frame in sequence, the signalcomprising a coded light signal; and obtaining a respective sample ofthe coded light signal from each of a number of said lines, said numberof lines being exposed over a duration, and said number being asubstantially smaller number of said lines than the plurality of linesexposed by the camera in each frame; wherein the coded light signalcomprises multiple repetitions of a message wherein the message has aduration longer than the duration over which said number of lines isexposed, and the message is repeated multiple times with a timing suchthat a different part of the message is seen by the camera in each of aplurality of different ones of said frames; and wherein the code isfurther configured so as when executed, it uses the frame duration andmessage duration to time align the different parts of the message fromthe plurality of different frames, and to reassembles the message fromthe time-aligned parts by performing at least the following steps: (i)for each of multiple frames, establish a sample per image line; (ii)collect all active samples of a given frame into a time-sequence, eachpositioned at the respective time at which the sample from that line waslocated within the frame, thereby forming a frame signal for each frame;(iii) time-aligning the frame signals across the message duration; (iv)reassembling the message from the time-aligned frame signals.
 17. Adevice comprising: an input for receiving a signal from arolling-shutter camera which captures frames of a given duration at agiven frame rate by exposing a plurality of lines of each frame insequence, the signal comprising a coded light signal; a signalprocessing module connected to said input and arranged to obtain, viasaid input, a respective sample of the coded light signal from each of anumber of said lines, said number of lines being exposed over aduration, and said number being a substantially smaller number of saidlines than the plurality of lines exposed by the camera in each frame;wherein the coded light signal comprises multiple repetitions of amessage wherein the message has a duration longer than the duration overwhich said number of lines are exposed, and wherein said multiplerepetitions of the message are received with a timing such that adifferent part of the message is seen by the camera in each of aplurality of different ones of said frames; wherein the signalprocessing module, based on the frame duration and message duration, isconfigured to time align the different parts of the message from theplurality of different frames; and reassemble the message from thetime-aligned parts by performing at least the following steps: (i) foreach of multiple frames, establish a sample per image line; (ii) collectall active samples of a given frame into a time-sequence, eachpositioned at the respective time at which the sample from that line waslocated within the frame, thereby forming a frame signal for each frame;(iii) extend the signals with zeros resulting in an extended framesignal, where the duration of each extended signal is n times themessage duration (n being an integer) and where the duration is longerthan the frame duration; (iv) time align the active samples by shiftingthe samples per line to thereby facilitate reassembly.